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Surround surveillance system for mobile body, and mobile body, car, and train using the 




same 





(57) A surround surveillance system mounted on a 
mobile body for surveying surrou ndings around the mo- 
bile body includes an omnlazimuth visual system, the 
omniazimuth visual system including: at least one om- 
niazimuth visual sensor including an optical system ca- 
pable of obtaining an image of 360° view field area 
therearound and capable of central projection transfor- 
mation for the Image, and an imaging section for con- 



verting the image obtained by the optical system into 
first image data; an image processor for transforming 
the first image data into second image data for a pano- 
ramic image and/or for a perspective image; a display 
section for displaying the panoramic image and/or the 
perspective image based on the second image data; 
and a display control section for selecting and control- 
ling the panoramic image and/or the perspective image. 
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(54) Method and apparatus for electronically distributing motion panoramic images 



(57) An electronic image distribution apparatus for 
distributing motion panoramic images. The main portion 
of the apparatus is a motion panoramic image server 
that has a source of motion panoramic images. The 
source of panoramic image frames may be a motion 
panoramic image playback device or a connection to a 
motion panoramic camera system. The motion pano- 
ramic image server transforms the panoramic image 
frames into an intermediate format that is more condu- 



cive for electronic transmission. In particular, portions of 
the panoramic image are encoded at a higher quality as 
compared to other portions of the panoramic image. A 
client computer obtains information about the encoding 
and subscribes to one or more tiles depending on the 
viewpoint of interest. The client computer receives and 
presents the data in these tiles. In addition, the applica- 
tion discloses how to adjust the subscription based on 
the actual and/or predicted pan operations to obtain a 
quality presentation. 
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Description 

[0001 ] The present invention relates to the field of im- 
age processing and data distribution. In particular the 
present invention discloses methods and apparatus for 
processing, transmitting, and post-processing pano- 
ramic image information. 

[0002] Most cam eras only record a view within a small 
viewing angle. Thus, a typical conventional camera only 
captures an image in the direction that the camera is 
aimed. Such conventional cameras force viewers to 
look only at what the camera operator chooses to focus 
on. 

[0003] Panoramic cameras capture a much wider 
field of field. A panoramic camera constructed using an 
optical "fish-eye" lens may capture at most a 210-de- 
gree field of view. A panoramic camera constructed with 
a catadioptric lens captures a full 360-degree panoramic 
image. With a large panoramic field of view, individual 
panoramic image viewers can decide what they will fo- 
cus on. 

[0004] Conventional images are often delivered by 
electronic means. For example, television and the Inter- 
net deliver conventional images across wired and wire- 
less electronic media. However, there are no standard 
means of delivering panoramic images electronically. 
Since panoramic images are so large, it is difficult to de- 
liver panoramic images using conventional image trans- 
mission techniques. To further compound the problem, 
motion panoramic images traditionally require a very 
high bandwidth channel for electronic distribution. Thus, 
it would be desirable to have an electronic distribution 
system that efficiently distributes motion panoramic im- 
age information. 

[0005] Another difficulty with motion panoramic imag- 
es is that a user often desires to rapidly pan the view 
from one area of the panorama to another. Because of 
the limited bandwidth available for sending the pano- 
ramic image from a motion panoramic image server to 
a client system, such rapid pan operations result in sig- 
nificant latency delays and/or temporal tearing between 
the time the user performs the pan operation and the 
display of the resulting view. Thus, it would be desirable 
to utilize the available bandwidth to provide both high- 
quality data (for the portion of the motion panoramic im- 
age being viewed) and lower-quality data (for the por- 
tions of the motion panoramic image that are not being 
viewed and dependent on the likelihood of that portion 
being viewed). 

[0006] Yet another problem with the prior art distribu- 
tion of motion panoramic images is caused by the laten- 
cy between a user's command and the result of the us- 
er's command. This latency is a function of the time re- 
quired to send the command to the image encoder and 
the time required by the encoder to propagate the re- 
sponse to the command back to the user. This latency 
can be of the order of tens of seconds. Thus, it would 
be advantageous to provide technology that allows the 



user's command to be satisfied by a server between the 
encoder and the client. 

[0007] Although the communication link between 
server computers and client computers are beginning to 

5 have more bandwidth with the use of digital subscriber 
lines (DSL) and ISDN, the amount of information flowing 
across these larger bandwidth links are more complex, 
are of higher resolution, and higher frame rates. Thus, 
bandwidth limitations occur even for these more ad- 

10 vanced communication links. Thus, it would be advan- 
tageous to provide techniques for using the available 
bandwidth to present a high-quality image while at the 
same time maintaining sufficient information about the 
non-presented portions of the panorama to minimize the 

*5 consequences of panning outside of the high-quality im- 
age. 

[0008] The present invention is defined in the append- 
ed claims. 

[0009] One embodiment comprises an electronic im- 
20 age distribution apparatus for distributing motion pano- 
ramic images. The main portion of the apparatus is a 
motion panoramic image server that has a source of mo- 
tion panoramic images. The source of panoramic image 
frames can be a motion panoramic image playback de- 
25 vice or a connection to a motion panoramic camera sys- 
tem. The motion panoramic image server transforms the 
panoramic image frames into an intermediate planar 
representation that is more conducive for electronic 
transmission. The motion panoramic image server then 
30 transmits the motion panoramic images to client sys- 
tems. 

[0010] In addition, one preferred embodiment disclos- 
es methods and apparatus for obtaining portions of a 
panorama over a limited bandwidth link. The invention 

35 uses an encoder array to multiply encode the panorama 
into tiles that have different characteristics. The compu- 
ter that uses the encoded data to present a view into the 
panorama subscribes to the tiles containing the encod- 
ed data. The tiles that are subscribed-to are selected 

40 based on the bandwidth available to transfer the tiles 
and the quality of the data in the tiles. The computer 
selects higher-quality tiles near the viewpoint and lower- 
quality tiles dependent on the distance the tile is from 
the viewpoint. As the viewpoint is altered (such as by a 

45 pan operation), the computer dynamically adjusts the 
tile subscription to present a quality image, if the pan 
operation moves the viewpoint such that data from low- 
er-quality tiles is required to present the view, the data 
is presented using the lower-quality data until the high- 

50 er-quality data resulting from the new tile subscription 
arrives. The presentation of the lower-quality data can 
be reduced or avoided by limiting the range of an allow- 
able pan, by predicting a future need for high-quality 
tiles based on a pan history, by providing tiered tiles near 

55 the viewpoint to reduce temporal and other presentation 
artifacts. 

[0011] Another preferred embodiment provides an 
entire panorama containing distributed-quality data to 
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the client computer. In this embodiment the panorama 
is encoded such that a portion of the panorama is en- 
coded for high-quality, other portions of the panorama 
are encoded for reduced-quality, and yet other portions 
of the panorama are encoded for low-quality (one skilled 
in the art will understand that more than three levels of 
quality can be used within the panorama). Multiple en- 
codings of the panorama differing by the placement of 
the relative quality portions are provided to the client. 
The client is thus able to select the panorama encoding 
that includes a high-quality portion responsive to the cli- 
ent's viewpoint. When the viewpoint moves away from 
the high-quality portion of the currently selected pano- 
rama encoding, the client subscribes to a different pan- 
orama encoding that includes a high-quality portion at 
responsive to the new viewpoint. 
[0012] Preferred embodiments include (without limi- 
tation) system methods, client computer side methods, 
server computer side methods, server apparatus, client 
apparatus, and computer program products for the serv- 
er computer and the client computer. 
[001 3] The invention will be described below with ref- 
erence to exemplary embodiments and the accompany- 
ing drawings, in which: 

Figure 1 illustrates one embodiment of a pano- 

ramic camera system that creates annu- 
lar images. 

Figure 2a illustrates an annular image that is re- 
corded by the panoramic camera system 
of Figure 1 . 

Figure 2b illustrates how the annular image of Fig- 
ure 2a appears after it has been un- 
wrapped by geometrical transformation 
software. 

Figure 3 illustrates the 360-degree band that the 
panoramic camera system of Figure 1 
captures and Figure 3 also illustrates a 
cylindrical projection data format. 

Figure 4 illustrates a networked computer sys- 
tems arrangement used to distribute mo- 
tion panoramic images captured by the 
panoramic camera system Figure 1 from 
a panoramic image server to client sys- 
tems. 

Figure 5 A illustrates a conceptual view of a spher- 
ical view captured by the panoramic 
camera system of Figure 1. 



10 



Figure 5C illustrates a panoramic image frame in a 
cylindrical intermediate planar represen- 
tation that has been divided into individ- 
ual tiles or a spherical intermediate pla- 
nar representation that has been adjust- 
ed and divided into tiles. 

Figure 5D illustrates a panoramic image frame in a 
low-resolution intermediate planar rep- 
resentation that has been divided into in- 
dividual tiles. 

Figure 5E illustrates a panoramic image frame in a 
medium resolution intermediate planar 
representation that has been divided into 
individual tiles. 

Figure 5F illustrates a panoramic image frame in a 
high-resolution intermediate planar rep- 
resentation that has been divided into in- 
dividual tiles. 

Figure 5G illustrates a panoramic annular image 
frame that has been divided into spatially 
variant tiles. 

Figure 6A illustrates a view within a panoramic im- 
age frame in an intermediate planar rep- 
resentation. 

Figure 6B illustrates four tiles from Figure 6A trans- 
mitted to construct a view within a client 
system. 

35 Figure 7 A illustrates a view within a panoramic im- 
age frame intermediate planar represen- 
tation that has suddenly shifted by a 
large angular value. 

to Figure 73 illustrates a view within a panoramic im- 
age frame intermediate planar represen- 
tation that has suddenly shifted by a 
small angular value. 

45 Figure 8 illustrates a flow diagram that describes 
how a panoramic image client system 
negotiates a connection with a motion 
panoramic image server. 

so Figure 9 illustrates a flow diagram that describes 
how a motion panoramic image server 
sends image tiles to a panoramic image 
client. 



15 



20 



25 



30 



Figure 5B illustrates a panoramic image frame in a 
spherical intermediate planar represen- 
tation that. has been unwrapped and di- 
vided into individual tiles. 



55 Figure 1 0 illustrates a flow diagram that describes 
how a motion panoramic image client 
renders a panoramic image view using 
image tiles received from a server. 
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Figure 11 illustrates a panoramic image frame in 
an intermediate planar representation 
that has been divided into overlapping 
tiles. 

Figure 12 illustrates a networked computer ar- 
rangement wherein a panoramic image 
server distributes motion panoramic im- 
age tiles on individual multicast chan- 
nels. 

Figure 1 3 illustrates a flow diagram that describes 
how a motion panoramic image client 
subscribes to multicast panoramic im- 
age tile channels and renders an image 
using received image tiles. 

Figure 14 illustrates a panoramic annular image di- 
vided into quadrants that are served on 
different multicast channels. 

Figure 15 illustrates a panoramic image server that 
transmits panoramic image information 
on different multicast channels on a mul- 
ticast backbone, and client systems that 
subscribe to the multicast channels. 

Figure 16A illustrates an overview of the architec- 
ture of one preferred embodiment. 

Figure 1 6B illustrates an apparatus to store a motion 
panorama image on a computer reada- 
ble media according to a preferred em- 
bodiment. 

Figure 1 6C illustrates an apparatus to read amotion 
panorama image from a computer read- 
able media according to a preferred em- 
bodiment. 

Figure 1 7 illustrates a computer system capable of 
using the invention in accordance with a 
preferred embodiment. 

Figure 1 8 illustrates a process of presenting a view 
into a panoramic source in accordance 
with a preferred embodiment. 

Figure 19 illustrates a conceptual example of 
tiered data used to present a panoramic 
image in accordance with a preferred 
embodiment. 

Figure 20A illustrates a process of sending tiles from 
a server in accordance with a preferred 
embodiment. 

Figure 20B illustrates a process of encoding tiles 



10 



15 



20 



25 



from a frame in accordance with a pre- 
ferred embodiment. 

Figure 21 illustrates application of an encoder ar- 
ray to a panorama in accordance with a 
preferred embodiment. 

Figure 22 graphical representation of one example 
tiling strategy that can be used with a 
preferred embodiment. 

Figure 23 graphical representation of a variable- 
quality encoding tier strategy that can be 
used with a preferred embodiment. 

[0014] A method and apparatus for compressing and 
distributing motion panoramic images is disclosed. In 
the following description, for purposes of explanation, 
specific nomenclature is set forth to provide a thorough 
understanding of the present invention. However, it will 
be apparent to one skilled in the art that these specific 
details are not required in order to practice the present 
invention. For example, the present invention's method 
of distributing panoramic images has been described 
with reference to the Internet. However, other types of 
digital information distribution systems (for example pri- 
vate data networks and satellite systems) may be used 
with the teachings of the present invention. 



30 The Basic Panoramic Camera Design 



[0015] Figure 1 illustrates an embodiment of one pos- 
sible 360-degree panoramic camera system. The image 
capture mechanism 150 of the panoramic camera sys- 
tem captures an angular image of the surrounding pan- 
orama that is reflected off the main reflector 110. More 
details about the panoramic camera system of Figure 1 
can be found in the copending patent application entitled 
"Panoramic Camera", filed June 11 , 1997, having serial 
number 08/872,525. 

[0016] Figure 2a illustrates an example of a panoram- 
ic image captured as an annular image. As illustrated in 
Figure 2a, the surrounding panorama is distorted by the 
main reflector having radial symmetry such that the sur- 
rounding represented as a two-dimensional annular im- 
age. In one embodiment, the shape of the main reflector 
transforms the standard rectangular coordinate system 
of the image into a polar coordinate system. To view the 
panoramic image captured as a two-dimensional annu- 
lar image, the annular image may be unwrapped to cre- 
ate an intermediate planar representation. The unwrap- 
ping is performed using a geometric transformation op- 
eration. Figure 2b illustrates how the annular image of 
Figure 2a appears after the images are geometrically 
transformed from the annular representation to a rec- 
tangular intermediate planar representation. 
[0017] The panoramic camera system of Figure 1 
captures an entire 360-degree image of the surrounding 
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horizon as illustrated in Figure 3. The panoramic camera 
system captures a panoramic image that includes ob- 
jects as low as 50 degrees below the horizon and ob- 
jects as high as 50 degrees above the horizon. However, 
alternate panoramic camera systems may capture a full 
spherical panoramic view, a partial 180-degree hemi- 
spherical view, or any other panoramic view. Further- 
more, a panoramic image may be constructed by com- 
bining a mosaic of conventional images captured with 
conventional camera systems. For example, eight con- 
ventional cameras can be used to capture images from 
eight different directions. Those images are then com- 
bined together to create a single panoramic image. 

A Motion Panoramic image Capture System 

[0018] When the image capture mechanism 150 of, 
..the panoramic camera of Figure 1 is a motion image 
capture mechanism such as a charged coupled device 
(CCD) or a film-based motion picture image capture sys- 
tem, then it is possible to capture a series of panoramic 
annular images. The series of panoramic annular imag- 
es can be used to generate a motion panoramic display. 
Figure 4 illustrates one possible embodiment of an elec- 
tronic motion panoramic image capture system. 
[0019] Referring to Figure 4, a panoramic camera 
system 405 captures a series of panoramic annular im- 
age frames with an electronic image capture mecha- 
nism. For example, the electronic image capture mech- 
anism can be a charged coupled device image capture 
mechanism or CMOS image sensor that generates dig- 
itized panoramic annular image frames. The panoramic 
annular image frames are transmitted from panoramic 
camera system 405 to a panoramic image server 400 
across a high bandwidth communication link 402. The 
communication link must be high bandwidth since each 
panoramic annular image frame contains a very large 
amount of information and : to provide a flicker-free mo- 
tion image, several panoramic images must be trans- 
mitted each second. 

[0020] Alternate embodiments of motion panoramic 
^image capture systems can also be used. For example, 
a motion panoramic capture system may be created 
with two motion picture camera systems pointing in op- 
posite directions wherein each motion picture camera 
system is equipped with a fisheye lens that captures at 
least a 180 degree hemispherical field of view. In such 
a panoramic image system, two distorted fisheye imag- 
oes from each hemispherical view would need to be 
"stitched" together to create a single panoramic image. 

A First Motion Panoramic Image Distribution 
System Embodiment 

[0021] As previously set forth, the full panoramic an- 
nular image frames generated by the panoramic camera 
system (such as panoramic camera system 405 of Fig- 
ure 4) are so large that a very high bandwidth link is 



needed to transmit full motion images to a recipient. 
Such high bandwidth links are rare and expensive. 
Thus, for mass distribution of panoramic imagery, more 
bandwidth efficient methods of distributing panoramic 

5 images are needed. 

[0022] The present invention introduces two different 
methods of distributing motion panoramic images. A first 
method divides panoramic images into tiles and trans- 
mitsonly the tiles needed to construct a desired image. 

10 A second method, described in a later section, divides 
panoramic images into individual tiles and then trans- 
mits all the tiles on different multicast channels. 

The Selective Tile Transmission System 

15 

[0023] To efficiently transmit a panoramic image, the 
present invention first proposes a selective tile transmis- 
sion system. In the selective tile transmission system, a 
server computer system divides each panoramic frame 
into tiles. The tile size and shape are preferably selected 
in such a manner that is dependent upon a coordinate 
system used by the format used to transmit the tiles. 
Then, the tiles needed to construct an image on a client 
computer system are transmitted from the server sys- 
tem to individual clients. 

[0024] In a preferred embodiment, the server system 
transforms each incoming panoramic image frame into 
an intermediate planar representation. This transforma- 
tion step is not necessary but provides certain advan- 
tages. The selective tile transmission system will be de- 
scribed with reference to a system that performs this 
transformation in Figures 4, 5, 6, 7, 8 and 9. 
[0025] Referring to the particular embodiment of the 
selective transmission system illustrated in Figure 4, a 
panoramic annular image transformation system 417 
"unwraps" annular panoramic image frames into an in- 
termediate planar representation. The intermediate pla- 
nar representation is more convenient for generating im- 
ages suitable for viewing. Specifically, the intermediate 
format should require fewer calculations to generate a 
final rectangular projection. 

[0026] In one embodiment, the incoming annularpan- 
oramic image data is geometrically transformed into a 
spherical coordinate system with a spherical projection 
representation. In such an embodiment, a single frame 
of the image data may be viewed with the Live Picture 
viewer from Live Picture, Inc. Figure 5A conceptually 
illustrates a spherical data format wherein the view of 
the surroundings is projected onto a sphere surrounding 
a viewpoint. 

[0027] In another embodiment, the incoming annular 
panoramic image data is geometrically transformed into 
acylindrical coordinatesystem in a cylindrical projection 
representation. An image transformed into a cylindrical 
coordinate system may be viewed with the QuickTime 
VR viewer created by Apple Computer, Inc. Figure 3 
conceptually illustrates a cylindrical data format wherein 
the view of the surroundings is projected onto a cylinder 
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surrounding a viewpoint. 

[0028] After converting the panoramic image frames 
into an intermediate planar representation, the system 
of the present invention divides the intermediate planar 
representation into individual tiles. In the present inven- 5 
tion, the size of the tiles are dependent upon the coor- 
dinate system of the intermediate planar representation. 
In such embodiments, the tiles may be spatially variant. 
[0029] The size of the tiles may further be dependent 
upon the client system that will be displaying the pano- io 
ramie image data. In such an embodiment, each client 
uses a different process (441 or 445) to divide interme- 
diate planar representations into tiles since each client 
may use different client sizes. However, in a standard- 
ized embodiment, the size of the tiles are fixed such that '5 
a single process (illustrated as dotted box 449) may be 
used to divide an intermediate planar representation in- 
to individual tiies. 

[0030] Figure 5A illustrates a conceptual diagram of 
the spherically encoded intermediate planar represen- 20 
tation. As illustrated in Figure 5A, the spherically encod- 
ed intermediate planar representation represents the 
surrounding world as image data mapped onto a 
sphere. To divide and organize the data, the sphere is 
broken into individual bands on sphere A, B, C, D, E, 25 
and F. Note that the bands near the "equator" of the 
spherically encoded intermediate planar representation 
contain more data than the bands closer to the "poles." 
[0031] In one embodiment of the present invention, 
the image data is represented by individual bands that 
are divided into individual tiles as illustrated in Figure 
5B. Note that there will be fewer tiles in the bands from 
higher inclination angles. Alternatively, there will be the 
same number of tiles, but the tiles will contain data from 
smaller yaw angles: If the data from the smaller yaw an- 
gles is increased, this type of intermediate data format 
can be graphically illustrated as shown in Figure 5C. 
Note that the graphical data representation of Figure 5C 
also matches the cylindrical data format of Figure 3. Al- 
though the data format of Figure 5B may be used and 
provides certain advantages (such as a more compact 
data format) this document will focus on the format of 
Figure 5C to simplify the description. 
[0032] Referring to the particular embodiment of Fig- 
ures 5A, 5B, and 5C, a first row of tiles covers from the 
horizon line to thirty degrees above the horizon. A sec- 
ond row of tiles covers from thirty degrees above the 
horizon to forty-two degrees above the horizon. A third, 
top, row of tiles covers the band from forty-two degrees 
above the horizon to fifty degrees above the horizon. 
The same angular divisions are used to divide the space 
below the horizon. The row size divisions are nonlinear 
and widest at the horizon since users are usually most 
interested in views near the horizon. However, different 
angular divisions, such as linear angular divisions, may 
also be used. 

[0033] After dividing each unwrapped panoramic im- 
age into tiles, a tile fetching program (440 and 443) 



fetches tiles for transmission to the client system. Ideal- 
ly, the tile-fetching program only selects the tiles needed 
to create a view on a client computer system. In a pre- 
ferred embodiment, the client computer system deter- 
mines which tiles are needed and sends a request that 
specifies these tiles. The fetched tiles are transmitted 
using a tile-streaming program (460 and 463). By trans- 
mitting only the tiles needed to create a view on the client 
computer system, the system of the present invention 
limits the amount of information that must be transmitted 
from a panoramic image source server to a panoramic 
image client display. 

Tile Tiering for Magnification 

[0034] To accommodate several different magnifica- 
tion levels, several different sets of tile resolution may 
be created. For example, a low resolution "zoomed 
back" initial set of tiles may be used for panning around 
a panoramic image. The low-resolution initial tiles would 
encompass a large angular view. When an interesting 
area of the panoramic image is selected, a higher res- 
olution set of tiles could then be selected. 
[0035] Figures 5D, 5E and 5F illustrate one possible 
embodiment where a panoramic image space has been 
divided into three tiers of magnification. An initial 
"zoomed back" tile magnification intermediate planar 
representation is presented in Figure 5D. The lower res- 
olution intermediate planar representation of Figure 5D 
is used first. When a user wishes to see a portion of a 
view in greater detail, the client software "zooms in" by 
fetching tiles from the next higher resolution intermedi- 
ate planar representation displayed in Figure 5E. If 
even greater resolution is needed, the client software 
"zooms in" further by fetching tiles from Figure 5F. In 
one embodiment, the pixel resolution of the individual 
tiles of Figures 5D, 5E and 5F is the same such that the 
individual tiles take up the same amount of screen space 
but provide successively greater resolution. 

Tiling without Transformation 

[0036] As stated in the beginning of this section, the 
initial panoramic image frames need not be transformed 
into an intermediate planar representation before divi- 
sion into individual tiles. The intermediate planar repre- 
sentation transformation is usually performed since the 
transformation is used to put the image into a format that 
can be used to quickly generate an image for display 
with minimal processing. However, the raw panoramic 
image frames may immediately be divided into tiles and 
transmitted. 

[0037] For example, Figure 5G illustrates a raw pan- 
oramic annular image frame from the panoramic cam- 
era system of Figure 1 . Before transmitting, the raw pan- 
oramic annular image is divided into tiles. In the embod- 
iment of Figure 5G, the raw panoramic annular image 
is divided into square tiles that are small near the center 
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since a small tile covers a wide image area in the annular 
format. 

[0038] The middle part of the annular disk is divided 
into larger tiles. The outside of the annular disk is divided 
into smaller tiles since a narrow strip of tiles is needed 5 
when constructing a view that uses information from the 
edge of the annular disk. Thus, the tiles are spatially var- 
iant. 

Tile Positional Selection 10 

[0039] Figure 6A illustrates an example of a client 
system view into a panoramic image. Specifically, client 
view 630 represents a view into an unwrapped pano- 
ramic image 610 requested by a client computer sys- 15 
tern. The view is distorted since the view is illustrated 
within the spherical coordinate space. To construct the 
view 630 at a client system, a server (such as server 
400) transmits tiles 621, 623, 625, and 627. Each tile 
621 , 623, 625, and 627 is compressed before transmis- 20 
sion to further reduce the amount of information that 
needs to be transmitted. Figure 6B illustrates how the 
four tiles 621 , 623, 625, and 627 transmitted from the 
server are used to construct the client view 630 on a 
client computer system. (The four tiles 621 , 623, 625, 25 
and 627 appear distorted since the tiles are displayed 
in the rectangular projection coordinate space of Figure 
6B) 

[0040] Referring back to Figure 6A, if a user moves 
the view 630 further to the right, then the view will cease 30 
to need tiles 621 and 623 and will instead begin to need 
tiles 628 and 629. To accommodate such situations, the 
client system begins requesting tiles that are just slightly 
out of a current view when a view nears the edge of a 
tile in case the user moves further in that direction. Thus, 35 
in the example of Figure 6A, the client system would be- 
gin requesting tiles 628 and 629 when the view shifts a 
little further to the right. Thus, the tile-fetching program 
/ in the server would fetch and transmit tiles 628 and 629 
to the client system. Then, if the client view 630 *o 
progresses even further to the right, tiles 623, 628, 627, 
and 629 would be used to construct the shifted client 
view in the client system. 

[0041 ] In a preferred embodiment, the client computer 
system is responsible for determining when the out of 45 
view tiles are needed and when they should be trans- 
mitted. The client system makes this determination by 
examining factors such as how fast the user is moving 
the view, the latency of the connection with the server, 
and the parameters that define the view. The client com- so 
puter system transmits a list of desired tiles to the server 
wherein the list of desired tiles defines which tiles should 
be transmitted by the server to the client computer sys- 
tem. 

[0042] The movement anticipation procedure de- 55 
scribed in the previous two paragraphs works well for 
small incremental client view shifts (which most view 
changes comprise). However, if a user makes a sudden 



quick view change, the needed tiles will not be available. 
For example, referring to Figure 7A, a user may quickly 
shift a view from a first view position 730 to a second 
view position 740. The first view position 730 was being 
rendered with tiles 721 , 723, 725, and 727 that were sent 
by a server. The new second view position 740 requires 
tiles 751, 753, 755, and 757 that were not being trans- 
mitted. 

[0043] To handle such sudden view changes, the 
present invention occasionally transmits a highly com- 
pressed version of the full panoramic image intermedi- 
ate planar representation. The highly compressed pan- 
oramic image intermediate planar representation is only 
transmitted once every 30 or so normal frames. When 
a fast movement occurs, the client renders a view from 
the most recently received highly compressed pano- 
ramic image intermediate planar representation. This 
newly rendered view will not be in the normal resolution 
since it is being rendered from the highly compressed 
panoramic image intermediate planar representation. 
Simultaneously, the server will begin transmitting the 
tiles needed at the new view location. 
[0044] In the example of Figure 7A, the server will be- 
gin transmitting tiles 751 , 753, 755, and 757. When tiles 
751 , 753, 755, and 757 are received by the client com- 
puter system, the image rendered from the most recent- 
ly received highly compressed panoramic image inter- 
mediate planar representation will be replaced with an 
image rendered from tiles 751 , 753, 755, and 757. Thus, 
the new view image will become clearer once the server 
"catches up" to the users sudden view change. 
[0045] A sudden movement may not move the view 
completely away from the currently available tiles. For 
example, a movement may place the view partially on 
tiles that are available and partially on tiles that are not 
available. In such a situation, the portion of the view that 
is located on available tiles can be rendered at the res- 
olution of those tiles and the remainder of the view will 
be rendered using the highly compressed version. Sim- 
ilarly, if the user is "zoomed in" on high resolution tiles 
and the user moves quickly away from the currently 
available high resolution tiles, the view may be con- 
structed using the tile from the next lower resolution tier 
if that tile is still available. 

[0046] Figures 8 and 9 provide detailed flow diagrams 
that fully describe how the electronic image distribution 
system of Figure 4 operates. Figure 8 illustrates a flow- 
chart describing one embodiment wherein a panoramic 
motion image server system and a client system nego- 
tiate a connection. However, it should be noted that in 
other embodiments, a standard connection can be used 
such that there is no negotiation of connection parame- 
ters. Figure 9 describes how a server system transmits 
image tiles to a client system. 
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Client-Server Motion Panoramic Image Connection 
Negotiation 

[0047] Figure 4 illustrates one embodiment of a mo- 
tion panoramic image client-server system wherein the 
bandwidth is conserved by dividing panoramic images 
into tiies. It should be noted that there is often not a 
"standard" server and client environment. Specifically, 
the clients, the server, and the network in a client-server 
environment may all vary depending on the circum- 
stances. The capabilities of each different panoramic 
server system and each different panoramic client sys- 
tem would vary depending upon the system's specifica- 
tions including the computer make, the processor type, 
the processor generation, the amount of random access 
memory available, the bus speed, and the operating 
system. 

[0048] To most efficiently distribute motion panoramic 
images from a motion panoramic image server to cli- 
ents, the present invention conducts a client-server ne- 
gotiation. The server system determines the character- 
istics of the client before determining how the client- 
server system will operate. Figure 8 illustrates a flow di- 
agram describing the method of negotiating the client- 
server motion panoramic image connection. 
[0049] Referring to Figure 8, a client initiates a con- 
nection to a panoramic image server at step 810. Next 
at step 820, the panoramic image server requests infor- 
mation that describes the display characteristics of the 
client. Specifically, the server may request the client's 
frame buffer size, the client's frame buffer color charac- 
teristics, the size of the panoramic image view window, 
and other information relevant to the display of images 
on the client system. The motion panoramic image serv- 
er may also request information describing the process- 
ing abilities of the client. For example, the panoramic 
image server may request the processor type, the proc- 
essor speed, and the amount of main memory available. 
The client responds to the server's by sending this in- 
formation to the motion panoramic image server at step 
830. 

[0050] At step 840, the motion panoramic image serv- 
er uses the collected information to determine how the 
server will serve the client. One specific parameter that 
the server must decide is how to divide the unwrapped 
panoramic intermediate planar representation into tiles. 
Specifically, the size of the individual tiles will be deter- 
mined. The motion panoramic image server may also 
determine the video frame-rate, a type of compression 
to use, and a processing-task division based upon the 
network connection speed and quality, the capabilities 
of the client system, and the current and anticipated 
processing load of the image server. 
[0051] After determining the parameters of the motion 
panoramic image connection, the motion panoramic im- 
age server transmits the connection parameters to the 
client system at step 850. The client system will use the 
parameters to determine how the client system will dis- 



play motion panoramic images. 

[0052] After the client and server have negotiated the 
motion panoramic image connection, the client begins 
requesting motion panoramic image information from 
5 the server system at step 860. In response to the re- 
quests, the server begins transmitting motion panoram- 
ic image tiles to the client system at step 870. Since the 
connection to each client is uniquely calibrated, the 
server will transmit panoramic image information to 
each different client in a manner that is specific to that 
particular client. The server will continue transmitting 
motion panoramic images until the client requests the 
server to stop sending images at step 880. 

Motion Panoramic Image Server Operation 

[0053] Figure 9 describes how the motion panoramic 
image server transmits the information needed to con- 
struct a view within a motion panoramic image on a cli- 
ent system. Thus, Figure 9 describes in detail what oc- 
curs during the steps 860 and 870 of Figure 8. 
[0054] Referring to step 910 of Figure 9, the first step 
that the motion panoramic image server must perform 
is transforming a panoramic image frame into one or 
more intermediate planar representations. The exact 
method of transforming a panoramic image frame is de- 
pendent on the type of camera system used. For exam- 
ple, a system built using the panoramic camera system 
of Figure 1 would perform a geometric transformation to 
transform the annular panoramic image frame into the 
selected intermediate planar representation. If a f isheye 
lens panoramic camera system were used, a different 
geometric transformation would transform the distorted 
fisheye image into the desired intermediate planar rep- 
resentation. All subsequent actions are performed on 
the transformed intermediate planar representation. 
[0055] In an embodiment with magnification, the 
source panoramic image data is transformed into more 
than one intermediate planar representation of different 
resolutions. For example, the source panoramic image 
data can be transformed into low, medium, and high- 
resolution intermediate planar representations as illus- 
trated in Figures 5D, 5E and 5F. 
[0056] After the panoramic image frame has been 
transformed into an intermediate planar representation, 
the method proceeds to step 920. At step 920, the meth- 
od determines if a highly compressed version of the full 
panoramic image frame should be transmitted. As pre- 
viously set forth, a highly compressed version of the full 
panoramic image frame is transmitted occasionally to 
handle sudden movements of the client's view. In one 
embodiment, the motion panoramic image server trans- 
mits a highly compressed version of the full panoramic 
image frame during first and every subsequent 
frame. Thus, during the first and every N th subsequent 
frame, the method proceeds to step 970 where the full 
panoramic image frame is compressed. Then, at step 
980, the motion panoramic image server begins trans- 
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mitting the highly compressed version of the full pano- 
ramic image frame. The transmission of the highly com- 
pressed version of the full panoramic image frame may 
occur all at once. Alternatively, the highly compressed 
version of the full panoramic image frame may be trans- 
mitted concurrently along with individual image tiles 
such that individual image tiles are not delayed. 
[0057] Next, at step 930, the one or more intermediate 
planar representation frames are divided into individual 
tiles as illustrated in Figures 5D, 5E and 5F. The size of 
the individual tiles is the size that was negotiated during 
the connection process described in the flow diagram of 
Figure 8. 

[0058] After the frame has been divided into individual 
tiles, the server then fetches and transmits the tiles 
needed to create the client's view at step 940. Each tile 
is compressed before transmission to conserve band- 
width and decrease latency. The client will use the trans- 
mitted tiles to construct an image. 
[0059] At step 950, the server method determines if 
the client system is done viewing the motion panoramic 
image. If the client is not done, the method proceeds 
back to step 960. 

Motion Panoramic Image Server Operation 

[0060] Figure 10 illustrates a flow diagram describing 
a method that a client system may follow to receive in- 
formation and render an image. It should be noted that 
Figure 10 only illustrates one possible embodiment of a 
client system and that significant variations can be cre- 
ated. For example, several of the steps described in Fig- 
ure 10 such as receiving tiles and rendering an image 
may be performed concurrently. Alternatively, less 
processing could be performed in the client system and 
more processing could be performed in the server. 
[0061] Referring to step 1010 of Figure 1 0, the client 
receives any and all requested panoramic image tiles 
and highly compressed full panoramic images transmit- 
ted by the server. At step 1020, the client attempts to 
render an image for the current view position. Ideally, 
the view will be rendered with the tiles needed for the 
view. However, if the client does not have the desired 
tiles, then the client renders an image of the current view 
using the available tiles for the view, tiles from lower res- 
olution tiers, and the most recently received version of 
the highly compressed full panoramic image. 
[0062] After rendering an image frame, the client sys- 
tem accepts input from a user at step 1030. The client 
system may accept input, such as directional com- 
mands from a computer mouse, that pans the view with- 
in the panoramic image. The client may also accept 
commands that zoom-in and change the vertical viewing 
angle. 

[0063] The client system then processes the user in- 
put to the server system at step 1030 to determine a 
next view position. At step 1040, the client system de- 
termines if the input specifies that the user is done view- 
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ing the panoramic image information. The client pano- 
ramic image-rendering program will terminate when the 
user no longer wishes to view the panoramic image. 
[0064] At step 1050, the client system determines the 

5 frame tiles that are needed to create the new view po- 
sition. The client system also determines if it should re- 
quest tiles that are just out of the current view but are 
likely to be needed in the future. Specifically, the client 
determines if the current view is near the edge of a tile. 

10 |f the client's view is near the edge of a tile, then the 
client will request tiles that are just outside of the client's 
current view. The tiles just out of view are requested 
such that if the client moves the view beyond the edge 
of the tile, then the client will be able to create a view 

*5 without having to request and wait for the server to then 
send the needed tile. Thus, the client anticipates small 
view movements and is prepared to generate a view. 
However, if the user makes a sudden large view change, 
then the client rendering engine can always fall back to 

20 the highly compressed version of the full panoramic im- 
age. 

[0065] Furthermore, at step 1050, the client system 
determines if any currently requested tiles are no longer 
needed. Tiles that are no longer needed consume val- 

25 uable bandwidth such that the client system should un- 
subscribe from tiles that are no longer needed. 
[0066] After determining the tiles needed for the cur- 
rent view, the client system determines at step 1060 if 
the needed tiles have already been requested and if 

30 there are no unnecessary tiles on request. If the needed 
tiles have already been requested and there are no un- 
necessary tiles on request, then the client system does 
not need to change the tile subscriptions. However, if 
new tiles are needed or currently subscribed tiles are no 

35 longer needed, then the client system sends a request 
to the server at step 1 070 that changes the tile subscrip- 
tion. 

[0067] At step 1080, the client system determines if 
any new frame tiles or compressed panoramic image 

40 frames have been received. If new frames have been 
received, then the method proceeds back to step 1010 
to accept the new frames. Otherwise, the method pro- 
ceeds back to step 1020 to render a new view image 
using the currently available tiles and the highly com- 

45 pressed full panorama image information. A different im- 
age may be generated from the same frame information 
if the user has shifted the view. 

Overlapping Tile embodiment 

50 

[0068] In one proposed embodiment, the panoramic 
image intermediate planar representation is divided into 
overlapping tiles. An example of this is provided in Fig- 
ure 11. By dividing the intermediate planar representa- 
tion into overlapping tiles, a degree of hysteresis is pro- 
vided such that small back and forth view changes will 
not cause the client system to oscillate requests for new 
tiles. 
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A Second Motion Panoramic Image Distribution 
System Embodiment 

[0069] The selective tile transmission system de- 
scribed in the previous section provides a useful method 
of delivering motion panoramic image information in a 
unicast environment where each client receives a 
unique individual data stream. However, since the mo- 
tion panoramic server must send out a unique data 
stream to each client, there will be a significant amount 
of redundant information being transmitted. To reduce 
the amount of redundant information being transmitted, 
the present invention also introduces a second multicast 
motion panoramic image serving system. 
[0070] Figure 12 illustrates a multicast motion pano- 
ramic image serving system constructed according to 
the teachings of the present invention. In the system of 
Figure 12, a motion panoramic camera system 1205 de- 
livers a series of high-resolution panoramic images 
across a high bandwidth link 1 202 to a motion panoram- 
ic server 1200. A Panoramic Camera Interface 1210 
handles the communication with the motion panoramic 
camera system 1 205. A transformation unit 1217 trans- 
forms the raw panoramic image frames into an interme- 
diate planar representation. As previously set forth, two 
possible intermediate planar representations are cylin- 
drically encoded intermediate planar representation and 
spherically encoded intermediate planar representa- 
tion. 

[0071] After a panoramic image frame has been 
transformed into an intermediate planar representation, 
the intermediate planar representation is divided into in- 
dividual tiles by unit 1241 . After dividing the intermediate 
planar representation into individual tiles, a set of tile 
streaming units 1261, 1262, 1263, ... 126n transmit the 
tiles on multicast channels 1293 on a high bandwidth 
network1290. Specifically, each tile position is assigned 
its own multicast channel. 

[0072] To display a view within a motion panoramic 
image, each client system subscribes to the multicast 
channels that carry the tiles needed to construct the de- 
sired view. For example, client system 1295 needs the 
two tiles delivered by tile streaming units 1261 and 1262 
such that client system 1295 subscribes to the multicast 
channels transmitted by those tile streaming units. 
[0073] Figure 13 provides a flow diagram that de- 
scribes one possible method that a client system may 
follow to render a view from a motion panoramic server 
constructed according to the teachings of Figure 12. In- 
itially, at step 1310, the client system subscribes to the 
multicast panoramic tile channels necessary to render 
a view from an initial view location. Routers and switch- 
es that couple the client system to a main multicast 
backbone will perform the necessary operations in order 
to begin delivering the subscribed multicast panoramic 
tile channels. 

[0074] Next, at step 1 320, the client system will render 
an image of the current view using the information from 



the subscribed multicast panoramic tile channels. It 
should be noted that other client systems will share the 
same multicast channels such that bandwidth is con- 
served. 

5 [0075] At step 1330, the client system accepts input 
from the user. The client system may accept input, such 
as directional commands from a computer mouse, that 
pans and tilts the user's view within the motion pano- 
ramic image. 

[0076] After accepting the user's input, the client sys- 
tem analyzes the user's input. If the user has indicated 
that he is done viewing the motion panoramic image at 
step 1340, then the client system unsubscribes from all 
the multicast tile channels and will stop displaying im- 
ages as set forth in step 1370. 

[0077] If the user's input changes the position of the 
view within the motion panoramic image, then the client 
system determines the tiles that will be necessary to 
render the new view position at step 1350. Specifically, 
referring back to Figure 5, the client system determines 
which tiles comprise the current view location. At step 
1360, the client determines if only the same tiles that 
are already subscribed to are needed. If so, then the 
method proceeds back to step 1320 to render a subse- 
quent image view frame. 

[0078] If different tiles are needed, the client system 
proceeds to step 1310. At step 1310, the client system 
subscribes to the multicast tile channels needed to 
render the current view. 

[0079] To limit the number of multicast channels used, 
the number of tiles that the panoramic image is divided 
into should be kept small. Figure 14 illustrates one pos- 
sible way of dividing the original source panorama. In 
the embodiment of Figure 14, the source panorama is 
divided into four quadrants: an upper quadrant, a left 
quadrant, a right quadrant, and a lower quadrant. (The 
quadrants are illustrated using dotted lines.) Each quad- 
rant becomes the source of a multicast channel. Thus, 
only four multicast channels are required. 
[0080] To handle pans of the user's view, the informa- 
tion supplied for each quadrant should overlap the other 
quadrants such that only one multicast channel needs 
to be subscribed to. Ideally, the overlap area will be at 
least as large as a client view. Referring to Figure 1 4, a 
set of overlapping quadrants are displayed: overlapping 
upper quadrant 1410, overlapping left quadrant 1420, 
overlapping lower quadrant 1430, overlapping right 
quadrant 1440. 

[0081] If very fast pans occur, the multicasting embod- 
iment may also include a highly compressed full pano- 
ramic image channel. The highly compressed full pan- 
oramic image channel would occasionally transmit a 
highly compressed version of the full panoramic image 
such that a view in any direction could be created. 
[0082] To divide the processing that must be per- 
formed, multiple servers may be used. For example, a 
first server could generate and serve the multicast chan- 
nel carrying the overlapping upper quadrant 1410, a 
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second server could generate and serve the multicast 
channel carrying the overlapping left quadrant 1420, 
and so on. 

[0083] Figure 15 graphically illustrates how network 
bandwidth is conserved using a multicast type of sys- 
tem. Specifically, a panoramic image server transmits 
all the tile channels on a multicast backbone. Various 
routers coupled to the multicast backbone only route 
multicast channels subscribed to by clients connected 
.4o their subnetworks. By routing only the multicast chan- 
nels needed by clients connected to their subnetworks, 
.the routers limit the tile information carried by the sub- 
networks. 

Tiered Quality Tiles 

[0084] This aspect of the invention addresses one dif- 
ficulty related to techniques for panning a view over a 
motion panorama. As the view is panned over the pan- 
orama (for example, by a user manipulating a control), 
:a tradeoff occurs between limiting the maximum allow- 
able pan rate and the amount of data that is kept to 
render the view when shifted by the pan operation. If 
unlimited bandwidth is available, the entire panorama 
could be delivered to the client computer for every frame 
in the motion panorama, there would be no need to limit 
the pan rate, and the view would always be presented 
at the desired quality. 

[0085] The amount of bandwidth consumed to send 
such a panorama over an unlimited bandwidth link is a 
function of the resolution of the panorama, the size of 
the panorama and the frame rate for updating the pan- 
orama. 

[0086] Where the link has limited bandwidth (such 
that the entire panorama cannot be sent at the desired 
resolution, size and frame rate because of the limited 
bandwidth), techniques can be used to allocate the 
available bandwidth to provide a quality view into the 
panorama, while still making lower-quality portions of 
the panorama available if required for a pan operation. 
This approach can result in significant temporal and 
•edge distortion when the viewpoint is panned into the 
lower-quality portion of the received panorama. Some 
of these distortions occur when the view is being gen- 
erated using data from different quality tiles and can re- 
sult in temporal tearing and other presentation artifacts. 
[0087] The previously described techniques (such as 
occasionally transmitting a highly compressed (low- 
equality) version of the panoramic image) tend to waste 
the available bandwidth because data representing por- 
tions of the panorama that are not expected to be viewed 
ds still sent on the off-chance that the view could be 
rpanned to an unexpected portion of the panorama. 
Thus, if these portions of the panorama are not needed 
the bandwidth used to send them is wasted. 
[0088] A preferred embodiment allocates the availa- 
ble bandwidth such that the available tile quality is a 
function of the distance the tile is from the current view- 



point. The tile quality is determined by the amount of 
compression (or other characteristics of the tiles) ap- 
plied to each tile. Thus, tiles that are adjacent to the 
viewpoint and used to render the view are less com- 

5 pressed to create a high-quality presentation of the view. 
Other tiles will be more highly compressed (thus, only 
able to present a lower quality view) responsive to the 
distance the tile is from the viewpoint. 
[0089] One skilled in the art will understand that many 

io techniques can be used to create tiles of differing quality. 
Each "tier" consists of tiles that have a particular quality. 
The quality is determined by characteristics such as the 
resolution of the portion of the image represented by the 
data in the tiles of the tier, the frame rate for sending the 

15 data defined by a tile, the amount of compression ap- 
plied to the data in the tile, and the amount of the color 
information maintained in the tile's data, along with other 
characteristics. A tier set is a collection of tiers that are 
encoded so as to balance the quality of the presentation 

20 of the tiles within the tier set with the bandwidth required 
to send the tile data. The creation and use of "tiers" is 
subsequently described. 

[0090] In addition, one skilled in the art will under- 
stand that many techniques exist to allocate the band- 

25 width between two data channels. These include, but 
are not limited to, sending multiple data streams at the 
same time (such as a mixed signal having two different 
data channels within the available bandwidth), or time- 
slicing the data channels onto a single communication 

30 link. 

[0091] Figure 16A illustrates a computer system 
1600 that includes a network 1601 used by at least one 
client computer, such as a first client computer 1 603 and 
an n th client computer 1605, to communicate with a 

35 server computer 1607. The server computer 1607 re- 
ceives information from an encoder array 1609 over a 
high-speed link. The encoder array 1 609 receives a mo- 
tion panorama image input 1611 and compresses the 
motion panorama image input 1611 into a number of 

40 tiered tiles of differing quality as is subsequently de- 
scribed. 

[0092] One skilled in the art will understand that the 
high-speed link can be a network connection, an internal 
computer bus, or any technology for rapidly transferring 
45 data from the encoder array 1 609 to the server computer 
1607. 

[0093] The computer system 1 600 can also include a 
media reader 1 613 that can access tiles that are already 
encoded, tiered and tiled and that are resident on com- 
50 puter readable media such as a DVD media 1615 or oth- 
er suitable media. 

[0094] The operation of the computer system 1 600 is 
that the motion panorama image input 1611 is provided 
to the encoder array 1609 on a frame-by-frame basis. 
55 The encoder array 1 609 compresses the motion pano- 
rama image input 1 61 1 to generate the tier and tile struc- 
ture subsequently described and sends this information 
to the server computer 1 607. The server computer 1 607 
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receives subscriptions to the differing quality tiles from 
the first client computer 1603 and/or the n lh client com- 
puter 1 605. The tile subscriptions need not be the same 
for each of the clients. The server computer 1607 dis- 
tributes the subscribed-to tiles to the clients over the net- s 
work 1601. The server computer 1607 can also access 
prerecorded tiers and tiles using the media reader 1613 
to read a computer readable media such as the DVD 
media 1615. 

[0095] Figure 16B illustrates a DVD media creation 
system 1620 that receives a motion panorama image 
input 1621 at an encoder array 1623 to generate the 
tiered and tiled information that is transferred to a DVD 
drive 1625 configured to write the tiered and tiled infor- 
mation onto a DVD media 1627. One skilled in the art 
will understand that other suitable media and media 
drivers can be used other than the DVD drive 1 625 and 
the DVD media 1627. 

[0096] Figure 16C illustrates an apparatus 1630 that 
can read tiered and tiled information from a DVD media 
1631 using a DVD drive 1633 under control of a com- 
puter system 1635. The computer system 1635 can 
present the accessed information as is subsequently 
described. In addition other suitable media and media 
drivers can be used. 

[0097] Figure 17 illustrates a computer system, indi- 
cated by general reference character 1700, that incor- 
porates the invention. The computer system 1700 in- 
cludes a processor 1701 that includes a central proces- 
sor unit (CPU) 1703, a memory section 1705 and an in- 
put/output (I/O) section 1707. The I/O section 1707 is 
connected to a user interface 1711 , a disk storage unit 
1713 and a DVD/CD-ROM drive unit 1715. The DVD/ 
CD-ROM drive unit 1715 can read a CD-ROM medium 
1717 that typically contains a program and data 1719. 
The DVD/CD-ROM drive unit 1715 (along with the CD- 
ROM medium 1717) and the disk storage unit 1713 
comprise a file storage mechanism. Some embodi- 
ments of the invention include a network interface 1721 
that connects the computer system 1700 to a network 
1723. Portions of the program and data 1719 can be 
resident in a program memory 1725 for execution by the 
CPU 1703. A DVD medium 1727 contains tired and tiled 
motion image data 1729 that can be read by the DVD/ 
CD-ROM drive unit 1715 for presentation by the user 
interface 1711, other presentation device, or a display 
device. A client computer 1731 can also access tier and 
tile information through the network 1723. In addition, 
motion images can be collected by a video camera 1733 
and processed by a video encoder array 1 735 to directly 
create the tier and tile information in real time. One 
skilled in the art will understand that not all of the dis- 
played features of the computer system 1700 are re- 
quired in some of the embodiments. 
[0098] Figure 18 illustrates a presentation process 
1800 that can be used by a computer to present a view 
into a panorama. The presentation process 1800 initi- 
ates at a 'start terminal 1801 and continues to a 'nego- 



tiate communication' procedure 1803. The 'negotiate 
communication' procedure 1803 communicates with the 
source of the panoramic image to determine the band- 
width available for transferring data representing the 
panorama to the computer. In one preferred embodi- 
ment this process determines the expected bandwidth 
between a networked panoramic image source and a 
client computer executing the presentation process 
1800. In a second preferred embodiment the 'negotiate 
communication' procedure 1803 determines the availa- 
ble bandwidth (communication speed) between a de- 
vice (such as the DVD/CD-ROM drive unit 1715 or the 
media reader 1613), used to access media containing 
the panoramic image, and the computer executing the 
'negotiate communication' procedure 1803. 
[0099] A 'receive availability information' procedure 
1 805 obtains availability information from the panorama 
source that represents the available tiers and tiles that 
can be provided by the panorama source. A 'subscribe 
to tiles' procedure 1 807 uses the availability information, 
the viewpoint into the panoramic image, and the band- 
width (as found by the 'negotiate communication' pro- 
cedure 1803) to determine a tile subscription for the 
computer. The 'subscribe to tiles' procedure 1807 can 
also specify that not all frames are to be sent for a sub- 
scribed-to tile (thus, skipping frames for that tile). A 're- 
ceive tiered tiles' procedure 1 809 receives the data from 
the subscribed-to tiles. This data is used to present a 
view into the panorama by a 'present view' procedure 
1811. 

[01 00] The presentation process 1 800 continues to a 
'pan operation' decision procedure 1813 that detects 
whether the viewpoint into the panorama has changed 
(for example, by a "pan" operation) or whether the view- 
point has remained substantially constant for a period 
of time (thus implying that the user is interested in the 
presented view and is relatively less likely to pan beyond 
the corresponding tile(s) representing the current view). 
[0101] If the viewpoint has not changed, the presen- 
tation process 1800 continues to an 'adjust subscription 
for quality* decision procedure 1815 that can adjust the 
tile subscription to increase (or reduce) the quality of the 
presented view. An increase in quality can be accom- 
plished by selecting a less compressed tile, by increas- 
ing the frame rate of the tile, by increasing the amount 
of color information provided with the original data (the 
color depth), by increasing the presented resolution, by 
increasing the size of the view, by adjusting the field of 
view, and/or by other mechanisms well understood by 
one skilled in the art. If the 'adjust subscription for qual- 
ity' decision procedure 1815 determines that the quality 
need not be adjusted, the presentation process 1800 
continues to the 'receive tiered tiles' procedure 1809. 
However, if the quality is to be adjusted, the presentation 
process 1800 continues to the 'subscribe to tiles' proce- 
dure 1807 that adjusts the tile subscription accordingly 
[01 02] However, if the viewpoint changed at the 'pan 
operation' decision procedure 1813, the presentation 
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process 1800 continues to a 'present new view* proce- 
dure 1817. The 'present new view* procedure 1817 
presents a new view based on the new viewpoint and 
possibly using reduced-quality tiles. 
[01 03] The presentation process 1 800 then continues 
to a 'significant pan operation' decision procedure 1819 
that determines whether the viewpoint change caused 
the view to be presented using data from reduced-qual- 
ity tiles (or is likely to do so in the near future). If so, the 
pan operation was a "significant pan operation" and re- 
quires a change to the tile subscription so as to receive 
high-quality tiles containing data for the presented view. 
In this instance, the presentation process 1800 contin- 
ues to the 'subscribe to tiles' procedure 1807 to make 
this change. 

[0104] However, if the 'significant pan operation' de- 
cision procedure 1819 determines that the pan opera- 
tion was not significant, the presentation process 1800 
continues to the 'receive tiered tiles' procedure 1809 to 
receive the data from the subscribed-to tiles for the next 
frame. 

[0105] The 'significant pan operation' decision proce- 
dure 1819 can be implemented in various ways. In one 
preferred embodiment, the 'significant pan operation' 
decision procedure 1819 monitors whether the pan op- 
eration has moved the viewpoint such that the genera- 
tion of the view uses data from a tile belonging to a low- 
er-quality tier. Another preferred embodiment uses a 
predictor to predict that the viewpoint will soon require 
a tile belonging to a lower-quality tier to generate the 
view and thus, subscribe to higher-quality tiles for the 
expected view. Other prediction techniques known in 
the art can be used. 

[0106] The 'adjust subscription for quality* decision 
procedure 1815 monitors changes in the viewpoint. If 
the viewpoint remains in a high-quality tier for a sufficient 
period (that is, when the viewpoint becomes stable), the 
'adjust subscription for quality' decision procedure 1 81 5 
can change the tile subscription to optimize the band- 
width use to increase the quality of the presented view. 
Examples of this would be to subscribe to tiles that have 
a higherframe rate or to subscribe to tiles that are less 
compressed. 

[0107] The presentation process 1800 can also detect 
when the viewpoint is likely to move out of the area of 
the panorama that has high-quality tiles and thus likely 
to require the view to be generated from data in a tile 
belonging to a lower-quality tier. The viewpoint stability 
can be determined by monitoring the viewpoint's history, 
predicting that the viewpoint into the panorama will 
change and that this change would cause the view to 
be generated using lower-quality tiles. In addition, the 
stability can be determined by monitoring how much the 
viewpoint has changed over a period of time. Other tech- 
niques well known in the art can also be used. 
[0108] The 'present new view' procedure 1817 uses 
data from existing tiles to generate the new view after 
the viewpoint has changed. If the viewpoint has 



24 

changed such that tiles from lower-quality tiers are 
needed to generate the view, the 'present new view' pro- 
cedure 1817 presents the view from this data. This re- 
sults in a lower-quality presentation until the newly sub- 

5 scribed-to tiles belonging to a high-quality tier are re- 
ceived at the 'receive tiered tiles' procedure 1809. Al- 
though the view is always presented, the quality of the 
presentation may be temporarily reduced and may in- 
clude temporal and/or spatial tearing or other quality ar- 

10 tifacts. 

[0109] Figure 19 illustrates an example tile subscrip- 
tion 1 900 having a quality axis 1 901 and a tile angle axis 
1903. In this space, there are eight tiles in three tiers. 
Tler-3 tiles 1905, 1913, 1915, 1917, 1919 are the tiles 

'5 that have the least quality data as indicated by the wide 
hashing in these tiles. Tier-2 tiles 1907, 1911 have better 
quality than thetier-3 tiles 1905, 1913, 1915, 1917, 1919 
but less quality than a tier-1 tile 1 909. In this illustration, 
each of the tiles 1905, 1907, 1909, 1911, 1913, 1915, 

20 1917, 1919 subtends 45 degrees of the panorama. 
Thus, a 360-degree panorama is represented by the da- 
ta in the eight tiles 1905, 1907, 1909, 1911, 1913, 1915, 
1917, 1919. The tier-1 tile 1909 includes the highest 
quality data such that presentation of the view from the 

25 tier-1 tile 1909 provides the best quality. If presentation 
of the view requires data from the other tiles, the viewer 
of the presentation will possibly notice temporal and/or 
spatial tearing. One skilled in the art will understand that: 
multiple tiles can be within each tier; that different angu- 

30 lar widths can be used with each tile, and that the sub- 
scribed tiles can be distributed as has been previously 
described. 

[0110] Figure 20A illustrates a server process 2000 
that can be used by the server computer 1607 to distrib- 

35 ute tiles to one or more clients (for example, the n m client 
computer 1605). The server process 2000 initiates at 
the 'start' terminal 2001 and continues to a 'connect with 
client' procedure 2003 that recognizes when a client 
computer requests a connection with the server compu- 

^o ter 1607 and performs the functions required to satisfy 
this request. Once the server computer 1607 and the 
client have established communication, the server proc- 
ess 2000 continues to a 'negotiate communication' pro- 
cedure 2005. The 'negotiate communication' procedure 

45 2005 is the server side of the communication negotiation 
of the 'negotiate communication' procedure 1803 per- 
formed by the client and previously discussed with re- 
spect to Figure 18. 

[0111] A 'send availability information' procedure 
50 2007 sends availability information to the client compu- 
ter. The availability information includes the character- 
istics and quality of the available tiers and the available 
tiles within the tiers. The client uses this information dur- 
ing the 'subscribe to tiles' procedure 1 807 previously de- 
55 scribed. 

[0112] A 'receive encodings for frame' procedure 
2009 receives a tiered and tiled frame of data from the 
encoder array 1 609 or by reading the DVD media 1615. 
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This procedure continues for so long as frames are 
available from the panorama source or until otherwise 
terminated. The 'receive encodings for frame' procedure 
2009 provides a 'frame received' signal 2010 to indicate 
when the received tiles can be sent to a client. 
[01 13] A 'receive tile subscription" procedure 201 1 re- 
ceives the tile subscription from the client computer. In 
one preferred embodiment, the tile subscription for each 
client is used to determine which tiles that are received 
from the encoder array 1609 are to be sent to each cli- 
ent. In another preferred embodiment, the tile subscrip- 
tion is used to select which tiles are to be broadcast on 
the multicast channels. A 'send tiles 1 procedure 2013 
sends the subscribed-to-tiles to the clients that have 
subscribed to them. The subscribed-to-tiles are sent re- 
sponsive to the 'frame received' signal 2010. Thus, as 
new encoded data arrives from the encoder array 1609, 
tiles containing portions of that data are sent to the client 
computers according to the tile subscription information 
provided by the client. 

[0114] A 'subscription changed' decision procedure 
2015 detects whether a client has changed the tile sub- 
scription. If the tile subscription for a particular client (or 
multicast channel) has not changed, the server process 
2000 continues to the 'send tiles' procedure 2013 to 
send the subscribed-to-tiles for the next frame. Howev- 
er, if the client changed its tile subscription, the server 
process 2000 continues to a 'subscription terminated" 
decision procedure 2017 to determine if all tiles are un- 
subscribed. If all tiles have been unsubscribed, the serv- 
er process 2000 completes through an 'end* terminal 
2019. Otherwise, the server process 2000 continues to 
the 'receive tile subscription' procedure 2011 to update 
the tile subscription for the client. 
[0115] Figure 20B illustrates an encoding process 
2030 for encoding creating tiled tiers of panoramic 
frames. The encoding process 2030 starts at a 'start* ter- 
minal 2031 and continues to an 'initialization' step 2033 
that initializes each of the encoders in the encoder array 
1 609 to provide the tier and tile characteristics according 
to the availability information for the panorama. An 'iter- 
ation' step 2035 determines when all the frames from a 
moving panorama image have been encoded. After ail 
the frames have been encoded the encoding process 
2030 completes through the 'end' terminal 2037. If ad- 
ditional frames exist in the panorama, the encoding 
process 2030 continues to a 'receive panorama frame 
data' step 2039 that receives a frame of the panoramic 
image from an image source such as a video camera, 
computer generated raster, or other image data source. 
Once the frame is received, the encoding process 2030 
continues to an 'encode frame data' step 2041. 
[0116] The 'encode frame data' step 2041 provides 
processing for each tile in each tier. One preferred em- 
bodiment simultaneously processes each portion of the 
frame to generate the tiles in parallel. Other embodi- 
ments can assign multiple tiles to a specific encoder. 
Thus, 24 encoders are used to encode a panorama with 



three-tiers and eight tiles per tier. Other tier/tile combi- 
nations will be subsequently described with respect to 
Figure 22. 

[0117] The encoder array 1609 can be configured to 
5 provide different levels of quality for each encoded tile. 
These quality parameters include the amount of com- 
pression, the frame rate, the view size and the color 
depth for each pixel in the tile. The color depth repre- 
sents the number of bits per pixel that represent the 
10 color of the pixel (including monochrome pixels and 
black & white pixels). 

[0118] A 'send encoded tiles' step 2043 sends the da- 
ta for each tile to the server computer 1607 using a high 
speed communications link (for example, a computer 
is data bus, a high-speed network link, or other high-speed 
communications) where it is stored in the server's mem- 
ory according to the availability information by the 're- 
ceive encodings for frame' procedure 2009 of Figure 
20A. 

20 [0119] One skilled in the art will understand- that the 
encoder array 1 609 can have its own memory for storing 
the tiles prior to sending the tiles to the server computer 
1607. 

[0120] Figure 21 illustrates an application of an en- 

25 coder array to a panorama 2100 that illustrates how an 
electronic panorama image 2101 (that includes a first 
portion 2103, a second portion 2105, a third portion 
2107, and a fourth portion 2109) is encoded by an en- 
coder array 2111 in one preferred embodiment. 

30 [0121] The electronic panorama image 2101 can be 
portioned in other ways than indicated. This includes 
ways where the portions are of differing sizes and/or 
where the portions overlap. In addition, the portions 
need not cover the entire height of the panorama but 

35 can be a single tile as has been previously discussed. 
In this preferred embodiment, the encoder array 2111 
has three tiers and three groups: a first group of encod- 
ers 21 1 3, a second group of encoders 21 1 5, and a third 
group of encoders 21 17. Other preferred embodiments 

40 organize the encoder array differently. These embodi- 
ments can include more or less encoders, more or less 
tiers, and/or more or less groups of tiers. 
[01 22] One preferred embodiment of the encoder ar- 
ray consists of only the first group of encoders 2113. 

45 This encoder array has a tier 1 encoder for each of por- 
tion 2103, 2105, 2107, 2109 of the electronic panorama 
image 2101 that encodes a high-quality version of their 
respective portions. The tier 2 encoders encode an in- 
termediate-quality version of their respective portions. 

50 The tier 3 encoders encode a reduced-quality version 
of their respective portions. Thus, in this embodiment, 
the first group of encoders 2113 generates tiered en- 
codings for each of the portions of the panorama. These 
encodings are sent to the server for each frame of the 

55 panorama. The server then distributes the encodings to 
the client computers according to their respective tile 
subscriptions. 

[0123] Another preferred embodiment includes the 
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second group of encoders 2115 and the third group of 
encoders 2117. These encoders generate tiers that 
have different quality characteristics from the first group 
of encoders 2113. 

[01 24] One skilled in the art will understand that a cli- 
ent computer (for example, the first client computer 
1603) can receive information about each generated tile 
in every tier. Thus, the client computer can subscribe to 
tiles from any tier to receive data that can be used to 
provide the desired quality presentation while still pro- 
viding responsive pan capability. 
[0125] A binary tree structure can be used to repre- 
sent the tiers and tiles. One skilled in the art will under- 
stand that many other data structures in memory can 
also be used to organize the tiles. 
[0126] One preferred approach for organizing the 
tiers and tiles is to create a tier set that has character- 
istics suitable for a particular available bandwidth. Thus, 
a client can select the tier set that is optimized for the 
available bandwidth between the server and the client. 
[0127] Figure 22 illustrates an example tiling strategy 
2200 using multiple groups of tiers, overlapping tiles, 
and different quality characteristics. The example tiling 
strategy 2200 includes a first tier set 2201 , a second tier 
set 2203, and a third tier set 2205. The first tier set 2201 
includes a high-quality 45-degree tier 2207, an interme- 
diate-quality 45-degree tier 2209, and a 'reduced-quality 
45-degree tier' 221 1 . The second tier set 2203 includes 
a 'high-quality 90-degree tier 1 2213, a 'high-quality 
90-degree (offset) tier* 2215, and a 'reduced-quality 
90-degree tier 1 2217. The third tier set 2205 includes a 
reduced-quality 180-degree tier 2219 and a reduced- 
quality 1 80-degree (offset) tier 2221 . A first wrapped tile 
2223 and a second wrapped tile 2225 are tiles that wrap 
around the planar representation of the panorama. 
[0128] In the illustration of the example tiling strategy 
2200, each tier has a number of tiles. The amount of 
data that is within the tile (and thus, the amount of band- 
width consumed by the transmission of the tile per an- 
gular portion of the panorama) is indicated by the height 
of the tile. Thus, the shorter the tile, the more com- 
pressed (and lower quality) is the data in the tile. Thus, 
the high-quality tile 2227 has less compression with re- 
spect to intermediate-quality tiles 2229 that in turn has 
less compression than low-quality tiles 2231. The data 
in each tileof a tier can be refreshed at a particular frame 
rate. However, the frame rate can be different for each 
tier in the tier set (and/or different for each tile). In par- 
ticular, each tier generally has quality characteristics 
that are different from every other tier. Tiles from multiple 
tiers (in different tier sets) can be subscribed-to as de- 
sired. 

[0129] One skilled in the art will understand that mul- 
tiple tier sets need not be included in the tiling strategy 
and that a binary tree data structure or other data struc- 
tures can represent a tiling strategy. 
[0130] In one preferred embodiment, the data in each 
tile indicated in the example tiling strategy 2200 is pro- 



vided by a single encoder of the encoder array 1609. 
Other preferred embodiments share the encoder among 
multiple tiles so long as the frame rate requirements are 
met for encoding the tiles. 

5 [0131] The tiers in each set can be related to provide 
a unified compression/quality schema that is often opti- 
mized for a particular bandwidth. For example, the tiers 
in the first tier set 2201 provide three levels of quality 
corresponding to the three tiers. Conversely the tiers in 

10 the second tier set 2203 consist of two high-quality tiers 
(the 'high-quality 90-degree tier* 2213 and the 'high- 
quality 90-degree (offset) tier 1 2215) having overlapping 
tiles and the 'reduced -quality 90-degree tier* 2217. The 
tiles in the third tier set 2205 consist of the reduced-qual- 

15 ity 180-degree tier 2219 and the reduced-quality 
180-degree (offset) tier 2221. One skilled in the art will 
understand that here are many different possibilities for 
the tiling and tier strategy other than that illustrated by 
the example tiling strategy 2200. One preferred embod- 

20 iment uses a binary tree structure for each tier set. Yet 
another preferred embodiment uses a plurality of tier 
sets similar to the first tier set 2201 but each tier set be- 
ing optimized for use with a particular bandwidth. 
[0132] The client computer determines which tiles to 

25 select based on the availability information. The availa- 
bility information specifies the available tile and tier in- 
formation. The client computer then subscribes to the 
appropriate tiles responsive to the desired quality of the 
presentation and the available bandwidth. The avaiia- 

30 bility information can include the tile information, the tier 
information, the compression information, and/or other 
information used to define the tiles available to the client 
computer. 

[01 33] The client computer receives the availability in- 
35 formation at the 'receive availability information' proce- 
dure 1 805 described with respect to Figure 1 8. If the cli- 
ent computer wants the example tile subscription 1900 
illustrated in Figure 19 using the availability information, 
for the example tiling strategy 2200, the client computer 
40 would subscribe to the high-quality tile 2227 to corre- 
spond to the tier-1 tile 1909, the intermediate-quality 
tiles 2229 to correspond to the tier-2 tiles 1907, 1911, 
and the low-quality tiles 2231 to correspond to the tier- 
3 tiles 1905, 1913, 1915, 1917, 1919. One skilled in the 
45 art will understand that this is but one example of how 
the client computer would subscribe to tiles for presen- 
tation. 

[0134] Another possible subscription, for example, is 
to subscribe to a high-quality tile 2237, low-quality tiles 

so 2233 (or a 'low-quality tile' 2235) along with a 'low-qual- 
ity tile' 2239. Yet another possible subscription is to sub- 
scribe to the high-quality tile 2237, the low-quality tiles 
2233 (or the 'low-quality tile' 2235) and a 'high-quality 
tile' 2241 thus allowing for extending high-quality pan 

55 operations for pans moving past 90 degrees toward 45 
degrees. 

[0135] The 'adjust subscription for quality' decision 
procedure 1815 (briefly discussed with respectto Figure 
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18) can dynamically adjust the tile subscription based 
on the history of the viewpoint movement. For example 
if the viewpoint has not changed over some period of 
time, the 'adjust subscription for quality' decision proce- 
dure 1815 can unsubscribe from the lower-quality tiles 5 
that have not been used to present the view (thus mak- 
ing more bandwidth available) and can subscribe to tiles 
that will improve the quality of the view at the expense 
of the ability to smoothly pan through the panorama. If 
the viewpoint starts to change, the 'adjust subscription io 
for quality* decision procedure 1815 can again sub- 
scribe to the lower-quality tiles to restore the pan capa- 
bility. 

[0136] One skilled in the art will understand that the 
example tiling strategy 2200 can provide different qual- '5 
ity tiles optimized for different quality characteristics and 
that the 'adjust subscription for quality' decision proce- 
dure 1815 can dynamically select tiles to best meet the 
changing needs, use, and operating constraints to 
present the highest possible quality view into the pano- 20 
rama. 

[0137] Figure 23 illustrates a distributed-quality en- 
coding tier strategy 2300 that can be used to simplify 
the subscription process in the client computer. The dis- 
tributed-quality encoding tier strategy 2300 provides an- 25 
other method for utilizing the available bandwidth be- 
tween the server and client computers. In this embodi- 
ment, the panorama is not partitioned into multiple tiles. 
Instead, each tile contains data for the entire panorama. 
Each tile is available for subscription by the client com- 30 
puter. Each of the tiles is encoded such that the quality 
of the encoded data in the tile is a function of the angular 
distance the encoded data is from a viewpoint. Thus, for 
example, one tile uses a high-quality data encoding from 
0 degrees to 90 degrees; a medium-quality data encod- 35 
ing from 90 to 1 80 degrees and from 0 to -90 degrees; 
and a low-quality data encoding from 180 to 270 de- 
grees. Using this approach, the client selects a particu- 
lar tile responsive to the client's viewport. As the view- 
port changes, the client subscribes to a different tile that 40 
has the high-quality data encoding at the changed view- 
point. 

[0138] Figure 23 graphically indicates the distributed- 
quality encoding tier strategy 2300 that (in this example) 
includes four different tiles of the panorama (a first tile *s 
2301 , a second tile 2303, a third tile 2305, and a fourth 
tile 2307). Each of these tiles contains data that repre- 
sents the panorama at the same time. Each of these 
tiles is encoded using a distributed quality encoding 
such that the encoding for the tile is distributed over the so 
tile. Thus, different portions of the tiles have its data en- 
coded for high-, medium- and low-quality. For example, 
assume that the tiles are separated into four quadrants 
of 90 degrees (a first portion 2309, a second portion 
2311, a third portion 2313 and a fourth portion 2315). 55 
Looking now at the portions in the tiles, notice that the 
first portion 2309 of the first tile 2301 is the high-quality 
encoded data (as indicated by the dense cross-hatch 
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pattern). If the first portion 2309 represents quadrant 
0-90 degrees, then the data in the adjacent quadrants 
(the second portion 2311 and the fourth portion 2315) 
could be encoded to medium-quality and the data in the 
third portion 2313 (representing the data directly behind 
the viewpoint) can be encoded to low-quality. Thus, by 
receiving the selected tile the client receives portions of 
the panorama as high-quality, medium-quality and low- 
quality data without a need to subscribe to multiple tiles. 
This simplifies the subscription process within the client 
computer. 

[01 39] One skilled in the art will understand that some 
embodiments do not implement a full 360 degree arc of 
the panorama. 

[01 40] Each of the other tiles has the high-quality en- 
coded data in a different portion of the tile. Thus, the 
client computer is able to display high-quality images by 
selecting which tile to receive responsive to the speci- 
fied viewpoint. In addition, each set of tiles can be opti- 
mized for use with a particular bandwidth. 
[0141] One skilled in the art will understand that the 
tiles can use portions having different characteristics 
than those shown in Figure 23. In particular, the portions 
can extend for more or less than the 90 degrees shown; 
need not cover the entire vertical extent of the pano- 
rama, nor do the portions need to cover the same an- 
gular area as any other portion. Such a one also will un- 
derstand that the high-quality portions of the different 
data encodings of the panorama can overlap. 
[0142] Some of the advantages of the invention as 
embodied by the described embodiments are: 

• Bandwidth usage is maximized to provide the best 
quality view into the panorama based on the past 
and/or predicted movement of the viewpoint. 

• Pan operations are not delayed due to missing tiles. 

• Pan operations are much more likely to generate 
views using high-quality tiles. 

• Tier sets can be configured for use at a specified 
bandwidth between the client and server comput- 
ers. 

• The use of tier sets that use distributed quality en- 
codings simplifies the selection mechanism within 
the client and is very efficient for limited size pres- 
entations. 

[01 43] One skilled in the art will understand that many 
of the processes described herein can be implemented 
differently than described. In particular, procedure func- 
tions and procedure ordering can often be modified with- 
out departing from the scope of the invention. Such a 
one will also understand that these embodiments are 
also applicable to still panoramas and the invention is 
not restricted to motion panoramas. 
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[01 44] The foregoing has described methods and ap- 
paratuses for electronically distributing motion pano- 
ramic images. It is contemplated that changes and mod- 
ifications may be made by one of ordinary skill in the art 
to the materials and arrangements of elements of the 
present invention without departing from the scope of 
the invention, as defined by the appended claims. 

Claims 

1 . A computer controlled method for presenting a view 
into a panoramic image, said method including 
steps of: 

obtaining availability information related to said 
panoramic image, said availability information 
including tier information; 

determining a viewpoint into said panoramic 
image, said viewpoint defining said view; 



wherein each of said one or more tiles is encoded 
using a distributed quality encoding for that tile. 

5. A computer controlled method according to claim 2 
5 wherein the step of accessing further includes steps 

of: 

subscribing to said one or more tiles responsive 
to said viewpoint and said availability informa- 
10 tion to establish a tile subscription; and 

receiving said one or more tiles responsive to 
said tile subscription. 

*5 6. A computer controlled method according to claim 5 
further including steps of: 

panning said viewpoint; 

20 detecting that the step of panning was a signif- 

icant pan of said viewpoint; and 



receiving a frame of said motion panoramic im- 
age; 

simultaneously encoding a plurality of portions 
of said frame into a plurality of encoded por- 
tions in accordance with said availability infor- 
mation; 

providing said plurality of encoded portions re- 
sponsive to a selection of said availability infor- 
mation; and 

presenting said view responsive to said data. 

2. A computer controlled method for presenting a view 
into a panoramic image, said method including 
steps of: 

obtaining availability information related to said 
panoramic image, said availability information 
including tier information; 

determining a viewpoint into said panoramic 
image, said viewpoint defining said view; 

accessing one or more tiles, responsive to said 
tier information, said one or more tiles contain- 
ing data used to present said view; and 

presenting said view responsive to said data. 

3. A computer controlled method according to claim 2 
wherein each of said one or more tiles is encoded 
at a particular quality for that tile. 

4. A computer controlled method according to claim 2 



changing said tile subscription. 

25 7. A computer controlled method according to claim 5 
or 6 further including steps of: 

detecting that said viewpoint is stable; and 

30 changing said tile subscription responsive to 

the step of detecting. 

8. A computer controlled method according to claim 5, 
6 or 7 further including steps of: 

35 

detecting that said viewpoint is substantially 
constant over a specified period; and 

changing said tile subscription to increase a 
40 quality of said view as presented. 

9. A computer controlled method according to claim 7 
wherein said quality is selected from the set con- 
sisting of presented resolution, field of view, view 

45 size, frame rate, compression, and color depth. 

10. A computer controlled method according to claim 2 
wherein said tier information includes tile informa- 
tion. 

50 

11 . A computer controlled method according to claim 2 
wherein said tier information includes compression 
information. 

55 1 2. A computer controlled method according to claim 2 
wherein said panoramic image is a motion pano- 
ramic image. 
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1 3. A computer controlled method according to claim 5 
wherein the step of obtaining said availability infor- 
mation further includes accessing a server compu- 
ter. 

14. A computer controlled method according to claim 
13 wherein said server computer performs steps of: 

sending said availability information; 

receiving said tile subscription; and 

sending said one or more tiles responsive to 
said tile subscription. 

15. A computer controlled method according to claim 2 
wherein the step of obtaining includes accessing in- 
formation from a computer readable media. 

16. A computer controlled method for encoding a mo- 
tion panoramic image, said method including steps 
of: 

receiving a frame of said motion panoramic im- 
age; 

simultaneously encoding a plurality of portions 
of said frame into a plurality of encoded por- 
tions in accordance with availability informa- 
tion; and 

providing said plurality of encoded portions re- 
sponsive to a selection of said availability infor- 
mation. 

17. A computer controlled method according to claim 
16 wherein said availability information includes tier 
information. 

18. A computer controlled method according to claim 
1 6 further comprising storing said plurality of encod- 
ed portions in a memory in accordance with said 
availability information. 

19. A computer controlled method according to claim 
16 wherein said availability information includes tile 
information. 

20. A computer controlled method according to claim 
16 further including a step of repeating the steps of 
receiving, of simultaneously encoding and of pro- 
viding. 

21. A computer controlled method according to claim 
16 further including a step of sending one or more 
of said plurality of encoded portions to a server com- 
puter. 



22. A computer controlled method according to claim 
16 wherein said availability information represents 
a data structure in memory having a plurality of tiers. 

5 23. A computer controlled method according to claim 

22 wherein each of said plurality of tiers includes 
one or more tiles, each of said one or more tiles ref- 
erencing data encoded respective to a quality. 

10 24. A computer controlled method according to claim 

23 wherein at least one of said plurality of tiers is 
respectively associated with a tier quality. 

25. A computer controlled method according to claim 
is 23 wherein at least one of said one or more tiles is 

respectively associated with a tile quality. 

26. A computer controlled method according to claim 
23, 24 or 25 wherein said quality is selected from 

20 the set consisting of presented resolution, field of 
view, view size, frame rate, compression, and color 
depth. 

27. A computer controlled method according to claim 
25 16 wherein the step of simultaneously encoding is 

performed by an encoder array. 

28. A computer controlled method according to claim 
1 6 wherein said plurality of encoded portions is con- 

30 tained within one or more tiles, each of said one or 
more tiles including data from a 270 degree to 360 
degree arc of said motion panoramic image. 

29. The computer controlled method of claim 1 6 where- 
35 in each of said plurality of encoded portions is in- 
cluded with one of said one or more tiles respec- 
tively. 

30. A computer program comprising computer readable 
40 code means adapted to instruct a computer to per- 
form the steps of any one of the preceding claims. 

31 . A computer program product comprising a storage 
medium having recorded thereon a computer pro- 
gram according to claim 30. 

32. An apparatus having a central processing unit 
(CPU) and a memory coupled to said CPU for pre- 
senting a view into a panoramic image, said appa- 

50 ratus includes: 

a first access mechanism configured to obtain 
availability information related to said pano- 
ramic image, said availability information in- 
55 eluding tier information; 

a viewpoint determi nation mechanism config- 
ured to determine a viewpoint into said pano- 
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ramie image, said viewpoint defining said view; 

a second access mechanism configured to ac- 
cess one or more tiles, responsive to the view- 
point determination mechanism and said tier in- 
formation, said one or more tiles containing da- 
ta used to present said view; and 

a presentation mechanism configured to 
present said view responsive to said data ac- 
cessed by the second access mechanism. 

33. An apparatus according to claim 32 wherein each 
of said one or more tiles is encoded at a particular 
quality for that tile. 

34. An apparatus according to claim 32 wherein each 
: of said one or more tiles is encoded using a distrib- 
uted quality encoding for that tile. 

35. An apparatus according to any one of claims 32 to 
34 wherein the second access mechanism further 
includes: 

a subscription mechanism configured to sub- 
scribe to said one or more tiles responsive to 
said viewpoint and said availability information 
to establish a tile subscription; and 

a receiver mechanism configured to receive 
said one or more tiles responsive to said tile 
subscription determined by the subscription 
mechanism. 

36. An apparatus according to claim 35 further includ- 
ing: 

a pan mechanism configured to pan said view- 
point; 

a detection mechanism configured to detect 
that the pan mechanism performed a significant 
pan of said viewpoint responsive to the pan 
mechanism: and 

a subscription modification mechanism config- 
ured to change said tile subscription responsive 
to the detection mechanism. 

37. An apparatus according to claim 35, 36 or 37 further 
including: 

a stability determination mechanism configured 
to detect that said viewpoint is stable; and 

a subscription modification mechanism config- 
ured to change said tile subscription responsive 
to the stability determination mechanism: 



38. An apparatus according to claim 35, 36 or 37 further 
including: 

a detection mechanism configured to detect 
5 r that said viewpoint is substantially constant 

over a specified period; and 

a subscription modification mechanism config- 
ured to change said tile subscription to increase 
10 a quality of said view as presented, the sub- 

scription modification mechanism responsive 
to the detection mechanism. 

39. An apparatus according to claim 38 wherein said 
*5 quality is selected from the set consisting of pre- 
sented resolution, field of view, view size, frame 
rate, compression, and color depth. 

40. An apparatus according to any one of claims 32 to 
20 39 wherein said tier information includes tile infor- 
mation. 

41 . An apparatus according to any one of claims 32 to 

40 wherein said tier information includes compres- 
25 sion information. 

42. An apparatus according to any one of claims 32 to 

41 wherein said panoramic image is a motion pan- 
oramic image. 

30 

43. An apparatus according to any one of claims 32 to 

42 wherein the first access mechanism is further 
configured to access a server computer. 

35 44. An apparatus according to claim 43 wherein said 
server computer includes: 

a first sending mechanism configured to send 
said availability information; 

a subscription receiver mechanism configured 
to receive said tile subscription; and 

a second sending mechanism configured to 
send said one or more tiles responsive to said 
tile subscription. 

45. An apparatus according to any one of claims 32 to 
42 wherein the first access mechanism includes a 
media reader device configured to access informa- 
tion from a computer readable media. 

46. An apparatus having a central processing unit 
(CPU) and a memory coupled to said CPU for en- 
coding a motion panoramic image, said apparatus 
includes: 

a frame receiver configured to receive a frame 
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of said motion panoramic image; 

an encoder array configured to simultaneously 
encode a plurality of portions of said frame re- 
ceived from the frame receiver into a plurality 
of encoded portions in accordance with availa- 
bility information; and 

a distribution mechanism configured to provide 
said plurality of encoded portions responsive to 
a selection of said availability information. 

47. An apparatus according to claim 46 wherein said 
availability information includes tier information. 

48. An apparatus according to claim 46 wherein said 
availability information includes tile information. 

49. An apparatus according to any one of claims 46 to 

48 further comprising a storage mechanism config- 
ured to store said plurality of encoded portions in 
said memory in accordance with said availability in- 
formation. 

50. An apparatus according to any one of claims 46 to 

49 further including a repetition mechanism config- 
ured to repeatedly invoke the frame receiver, the 
encoder array and the distribution mechanism. 

51. An apparatus according to any one of claims 46 to 

50 wherein the encoder array further includes a 
communication mechanism configured to send one 
or more of said plurality of encoded portions to a 
server computer. 

52. An apparatus according to claim 46 wherein said 
availability information represents a data structure 
in memory having a plurality of tiers. 

53. An apparatus according to claim 52 wherein each 
of said plurality of tiers includes one or more tiles, 
each of said one or more tiles referencing data en- 
coded respective to a quality. 

54. An apparatus according to claim 54 wherein at least 
one of said plurality of tiers is respectively associ- 
ated with a tier quality. 

55. An apparatus according to claim 53 wherein at least 
one of said one or more tiles is respectively associ- 
ated with a tile quality. 

56. An apparatus according to claim 53, 54 or 55 where- 
in said quality is selected from the set consisting of 
presented resolution, field of view, view size, frame 
rate, compression, and color depth. 

57. An apparatus according toa ny one of claims 46 to 
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53 wherein said plurality of encoded portions is con- 
tained within one or more tiles, each of said one or 
more tiles including data from substantially a 360 
degree arc of said motion panoramic image. 

58. An apparatus according to any one of claims 46 to 
57 wherein each of said plurality of encoded por- 
tions is included with one of said one or more tiles 
respectively. 
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